Harmonic Analysis of Neural Networks

It is known that superpositions of ridge functions (single hidden-layer feedforward neural networks) may give good approximations to certain kinds of multivariate functions. It remains unclear, however, how to effectively obtain such approximations. In this paper, we use ideas from harmonic analysis to attack this question. We introduce a special admissibility condition for neural activation functions. The new condition is not satisfied by the sigmoid activation in current use by the neural networks community; instead, our condition requires that the neural activation function be oscillatory. Using an admissible neuron we construct linear transforms which represent quite general functions f as a superposition of ridge functions. We develop • a continuous transform which satisfies a Parseval-like relation • a discrete transform which satisfies frame bounds Both transforms represent f in a stable and effective way. The discrete transform is more challenging to construct and involves an interesting new discretization of time-frequency-direction space in order to obtain frame bounds for functions in L2(A) where A is a compact set of R. Ideas underlying these representations are related to Littlewood-Paley theory, wavelet analysis and group representation theory.